System and Method for Intelligent Camera Control

ABSTRACT

Systems and methods for controlling camera settings of a camera to improve detection of faces in an uncontrolled environment are described. A first image is received from the camera, where the first image is captured by the camera at a first set of camera settings. A face is detected in the first image. The camera is adjusted to a second set of camera settings based on the detected face, where the second set of camera settings different from the first set of camera settings. A second image is received from the camera, where the second image is captured by the camera at the second set of camera settings. The face is detected in the second image. A quality metric of the face in the second image is determined where the quality metric is indicative of an image quality of the face in the second image. The camera is adjusted to a new set of camera settings to increase the quality metric of the face in subsequent images, the new set of camera settings different from both the first set of camera settings and the second set of camera settings. Once a sufficient quality metric of the face is achieved, the face is acquired, or otherwise captured, by the camera or other sensors.

FIELD OF THE INVENTION

The invention is generally related to various settings of a video cameraand more particularly, to controlling such settings to improve detectionand recognition of potential targets in an uncontrolled environment.

BACKGROUND OF THE INVENTION

Cameras, such as, but not limited to, consumer cameras (including videocameras), cell-phone cameras, and other conventional cameras employcertain camera settings designed to achieve some overall image quality.These camera settings, which may include aperture, gain, exposure, andother camera settings, have a significant influence on a quality ofimages acquired by a camera as well as subsequent processing of thoseimages, for example, facial recognition, etc.

In some instances, conventional cameras employ face detection to assistthe camera in adjusting these camera settings. However, theseconventional cameras may sacrifice, for example, an exposure of abackground (e.g., either underexposure or overexposure of thebackground) in favor of an exposure of the detected face. Suchunderexposure or overexposure of the background may result in theconventional camera failing to detect other faces in a scene. Inaddition, even if multiple faces are detected in the scene, suchconventional cameras may have difficulty determining which detected faceto use for adjusting the camera settings or difficulty generating camerasettings sufficient for all faces.

The camera settings impact a quality of images acquired by the camera,which in turn, impact a performance of any subsequent image processingperformed on the acquired images. Such image processing may include facedetection, face recognition, or other computer vision algorithms. Insome instances, a “quality” of the acquired images for purposes of facedetection (or other image processing such as facial recognition) maydiffer from a “quality” of the acquired images for purposes ofaesthetics (i.e., human perception of “image quality”).

What is needed is an improved system and method for intelligent cameracontrol that enhances performance of subsequent image processing onacquired images.

SUMMARY OF THE INVENTION

Various implementations of the invention relate to systems and methodsfor intelligent camera control that enhances performance of subsequentimage processing on images acquired from a camera. In someimplementations of the invention, a first image is received from thecamera, where the first image is captured by the camera at a first setof camera settings. A face is detected in the first image. In someimplementations of the invention, the camera is adjusted to a second setof camera settings based on the detected face, where the second set ofcamera settings is different from the first set of camera settings. Insome implementations of the invention, a second image is received fromthe camera, where the second image is captured by the camera at thesecond set of camera settings. The face is detected in the second image.In some implementations of the invention, a quality metric of the facein the second image is determined where the quality metric is indicativeof an image quality of the face in the second image. In someimplementations of the invention, the camera is adjusted to a new set ofcamera settings to increase the quality metric of the face in subsequentimages, the new set of camera settings different from both the first setof camera settings and the second set of camera settings. In someimplementations of the invention, once a sufficient quality metric ofthe face is achieved, the face is acquired, or otherwise captured, bythe camera or other sensors.

These implementations, their features and other aspects of the inventionare described in further detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an image processing system according to variousimplementations of the invention.

FIG. 2 illustrates various components of an image processor according tovarious implementations of the invention.

FIG. 3 illustrates an operation of image processing system according tovarious implementations of the invention.

DETAILED DESCRIPTION

Detecting a target (e.g., a person or other target) or features of atarget (e.g., a face of a person or other feature) in an uncontrolledenvironment is challenging, especially in an uncontrolled outdoorenvironment. First, the target is free to move into, out of, and withina field of view of the camera, at a variety of ranges and any number ofother motion factors as would be appreciated. Second, illumination ofthe target differs by weather, time of day, orientation of the target,objects in the environment, and any number of other illumination factorsas would be appreciated. Third, illumination of various regions of thetarget (e.g., a face) may dramatically differ from other regions of thetarget or other areas in the field of view. Fourth, having the targetinside a vehicle dramatically increases the challenges by introducingvehicle type, vehicle motion, location of the target in the vehicle,window tinting, reflections, sunroofs, interior lighting, and any numberof other vehicle factors as would be appreciated. Other factors providefurther challenges to detecting faces in the uncontrolled environment.Developing a set of camera settings to provide an overall target forimage quality is challenging.

According to various implementations of the invention, a set of camerasettings is adjusted (i.e., changed from a first set of camera settingsto a second set of camera settings) based on a probability ofsuccessfully recognizing a face of a target in an uncontrolledenvironment. According to various implementations of the invention, aset of camera settings is adjusted based on a probability ofsuccessfully detecting a face of a target in the uncontrolledenvironment, and then the set of camera settings is adjusted again basedon a probability of successfully recognizing the detected face of thetarget. In some implementations of the invention, this may beaccomplished by determining a quality metric for the image, and moreparticularly, for the detected face in the image. According to variousimplementations of the invention, a set of camera settings is adjustedonce a windshield (or other window) is detected in the uncontrolledenvironment; then the set of camera settings is adjusted again based ona probability of successfully detecting a face of a target behind thewindshield; and then the set of camera settings is adjusted again basedon a probability of successfully recognizing the face of the targetbehind the windshield. In various ones of the foregoing implementationsof the invention, once a certain quality metric for the image isachieved, the face is acquired, by, for example, a lidar systemconfigured to generate a three-dimensional image of the face.

More generally speaking, according to various implementations of theinvention, a set of camera settings may be adjusted once some eventoccurs (e.g., a detection of some aspect of the image, etc.); and thenthe set of camera settings may be adjusted again based on the occurrenceof some other, and in some cases independent, event occurs (e.g., adetection of some other aspect of the image, etc.). As above, thisadjusting may be iterated any number of times based on the samedetection or different detections until some result is achieved. By wayof example, but not of limitation, a vehicle may be detected in animage, and the set of camera settings adjusted to improve detection, forexample, a license plate. Then, once a license plate is detected, theset of camera settings may be adjusted to improve recognition ofcharacters on the license plates. At any point in this process, the setof camera settings may be adjusted, iteratively adjusted, orcontinuously adjusted until some result is achieved, which in theexample of the license plate, a high resolution, high contrast image ofthe license plate sufficient to read the characters, either by person ormachine.

FIG. 1 illustrates an image processing system 100 according to variousimplementations of the invention. Image processing system 100 includes acamera 110 and an image processor 120. In some implementations of theinvention, camera 110 includes a video camera. In some implementationsof the invention, camera 110 includes a digital camera. In someimplementations of the invention, camera 110 includes a digital videocamera. Camera 110 captures and outputs, to image processor 120, animage stream 130 comprised of a series of image frames 135 (alsoreferred to herein as images 135). Some of these image frames 135 mayinclude a target, or a face of the target (not otherwise illustrated) aswould be appreciated.

In some implementations of the invention, camera 110 includes a videocamera that provides a plurality of images 135 of the target that worksin combination with a lidar system that provides a range measurementand/or a Doppler velocity measurement for each of a plurality of pointson the target (or its face). Such a combined video camera and lidarsystem is available from Digital Signal Corporation, Chantilly, Virginiaand described in U.S. Pat. No. 8,717,545 to Sebastian et al., which isincorporated herein by reference in its entirety.

In some implementations of the invention, image processor 120 maycomprise various hardware, software, firmware and/or any combinationthereof that may be configured to perform various functions, includingthe functions described herein, as would be appreciated. Once soconfigured, image processor 120 becomes a particular machine configuredto implement various features and aspects of the invention as would beappreciated. In some implementations of the invention, image processor120 includes a computing processor and a memory (not otherwiseillustrated), where the memory is configured to store instructions that,when executed by the computing processor, implement and/or performvarious features and aspects of the invention, again, as would beappreciated.

FIG. 2 illustrates image processor 120 according to variousimplementations of the invention. In some implementations of theinvention, image processor 120 includes a face detector 210, a settingsadjuster 220, a face recognizer 240, and a face acquirer 250. In someimplementations of the invention, image processor 120 includes facedetector 210, settings adjuster 220, a windshield detector 230, facerecognizer 240, and face acquirer 250.

Face detector 210 receives image 135 and detects a face in image 135 inaccordance with various known techniques for detecting faces in imagesof uncontrolled environments. In some implementations of the invention,face detector 210 outputs confirmation of a presence of a face in image135. In some implementations of the invention, face detector 210 outputsa relative location of the face in image 135. In some implementations ofthe invention, face detector 210 outputs a relative location and anextent of the face in image 135.

In some implementations of the invention, based on the presence,location, and/or extent of the face in image 135, settings adjuster 220adjusts various camera settings 155 of camera 110. These camera settings155 may include an aperture of camera 110, an exposure time of camera110, a gain of camera 110, a region of interest of camera 110, a focusof camera 110, a zoom of camera 110, a white balance of camera 110, aresolution of camera 110, a cropping of an image from camera 110, a perpixel bit depth of camera 110, and/or other camera setting as would beappreciated. In some implementations of the invention, settings adjuster220 changes camera settings 155 from a first set of camera settings 155to a second set of camera settings 155 based on the presence, location,and/or extent of the face in image 135.

In some implementations of the invention, face recognizer 240 receivesthe location and/or extent of the face in image 135 and determines alikelihood that the face will be recognized. In some implementations ofthe invention, face recognizer 240 determines a quality metric (e.g., acolor histogram, a sharpness, a feature strength, an image quality,etc.) or other objective measurement of an image quality of the face inimage 135. If face recognizer 240 determines, based on the qualitymetric, that the image quality of the face in image 135 is insufficientfor facial recognition, face recognizer 240 instructs settings adjuster220 to provide a new set of camera settings 155 to camera 110.

In some implementations of the invention, if face recognizer 240determines, based on the quality metric, that the image quality of theface in image 135 is sufficient for facial recognition, face recognizer240 instructs face acquirer 250 to capture or otherwise acquire anotherimage 135 of the face. In some implementations of the invention, faceacquirer 250 causes camera 110 to capture a high resolutiontwo-dimensional image of the face using the latest set of camerasettings 155. In some implementations of the invention, face acquirer250 causes a lidar system to scan and capture a three-dimensional imageof the face (i.e., a collection of motion-compensated, three-dimensionalmeasurements of the face). In some implementations of the invention,face acquirer 250 causes camera 110 to capture a high resolutiontwo-dimensional image of the face using the latest set of camerasettings 155 and causes the lidar system to scan and capture athree-dimensional image of the face. In some implementations of theinvention, due to time associated with capturing the three-dimensionalimage, face acquirer 250 instructs settings adjuster 220 to continuouslyadjust camera settings 155 for camera 110, as necessary to compensatefor changes that may occur during this time, while capturing thethree-dimensional image.

In some implementation of the invention, windshield detector 230receives image 135 and detects a windshield, or other window, in image135 using various image processing techniques as would be appreciated.By way of example, a rectangular region of an appropriate size may bedetected in image 135 using for example a Hough transform as would beappreciated. Also by way of example, windshields may be detected viamachine learning techniques by training on images that includewindshields as would be appreciated. In some implementations of theinvention, windshield detector 230 detects a windshield or other windowor transparent surface in accordance with techniques described inco-pending U.S. Patent Application No. ______ (Attorney Docket No. D1251310.1), entitled “System and Method for Determining Ranges to a TargetBehind a Transparent Surface,” and filed on even date herewith, which isincorporated herein by reference in its entirety.

In some implementations of the invention, windshield detector 230outputs confirmation of a presence of a windshield in image 135. In someimplementations of the invention, windshield detector 230 outputs arelative location of the windshield in image 135. In someimplementations of the invention, windshield detector 230 outputs arelative location and an extent of the windshield in image 135. In someimplementations of the invention, windshield detector 230 outputs arelative location, an extent, and a range of the windshield in image135.

In some implementations of the invention, based on the presence,location, and/or extent of the windshield in image 135, settingsadjuster 220 adjusts various camera settings 155 of camera 110. Once awindshield is detected, face recognizer 250 may determine an expectedlocation of a face behind the windshield and settings adjuster 220 mayadjust various camera settings 155 for that expected location and toaccount for the face being behind a windshield and hence, within apoorly or non-uniformly, lit vehicle.

FIG. 3 illustrates an operation 300 of image processor 120 in accordancewith various implementations of the invention. In an operation 310,settings adjuster 220 adjusts camera settings 155 to a predetermined setof camera settings. The predetermined set of camera settings areoptimized to provide a predetermined image quality of a full frame ofcamera 110 or some relevant portion thereof (i.e., some portions of theframe of camera 110 may simply not matter). Such camera settings 155 maybe optimized based on application, environment, time of day, or otherfactors as would be appreciated.

In some implementations of the invention, settings adjuster 220 mayiteratively adjust camera settings 155 through a number of predeterminedsets of camera settings, each set tuned to a different one of variousscenarios that might exist in the uncontrolled environment. Thesedifferent predetermined sets ensure that a face in the uncontrolledenvironment is detected by one of the predetermined sets when the facemight not be detected by another one of the predetermined sets.Iterating through different predetermined sets of camera settings inaccordance with various implementations of the invention permits imageprocessing system 100 to search for faces in the entire frame.

In an operation 320, face detector 210 receives an image 135 from camera110, which is set to one of the predetermined set of camera settings,and detects a face in image 135. Face detector 210 provides informationregarding the detected face to settings adjuster 220. In an operation330, settings adjuster 220 adjusts camera settings 155 to a “face” setof camera settings. The “face” set of camera settings are optimized toprovide a predetermined image quality of the face in image 135. In someimplementations of the invention, the “face” set of camera settings mayinclude a region of interest which may be set to the detected face inimage 135.

In an operation 340, face recognizer 240 receives an image 135 fromcamera 110, which is set to a “face” set of camera settings, detects theface, and determines a quality metric of the face in image 135, thequality metric indicative of an image quality of the face in image 135.In an operation 350, settings adjuster 220 iteratively sets camera 110to a new “face” set of camera settings based on the image quality of theface in image 135. In an operation 360, face acquirer 250 causes camera110 to acquire a high quality image of the face using the newest “face”set of camera settings once the quality metric achieves a sufficientlevel. In some implementations of the invention, settings adjuster 220continues to adjust camera settings 155 while camera 110 acquires thehigh quality image of the face.

In a first optional operation (not otherwise illustrated in FIG. 3),windshield detector 230 detects a windshield in the scene. As discussedabove, windshield detector 230 may detect such a windshield in image 135from camera 110 or may detect such a windshield based on measurementsfrom a lidar. Windshield detector 210 provides information regarding thedetected windshield to settings adjuster 220. In a second optionaloperation (also not illustrated in FIG. 3), settings adjuster 220adjusts camera settings 155 to a “windshield” set of camera settings. Insome implementations of the invention, the “windshield” set of camerasettings are optimized to provide a predetermined image quality of thewindshield in image 135. In some implementations of the invention, the“windshield” set of camera settings are optimized to provide apredetermined image quality of a potential face behind windshield inimage 135. In some implementations of the invention, the “windshield”set of camera settings may include a region of interest which may be setto the detected windshield in image 135. In some implementations of theinvention, operation 300 continues with face detector 210 receiving animage 135 from camera 110, which is set to a “windshield” set of camerasettings, and detecting a face in image 135.

According to various implementations of the invention, each of the setsof camera settings (e.g., predetermined sets, “face” sets, “windshield”sets, etc.) may be determined based on various training scenarios tooptimize performance of image processing system 100 in each of thevarious stages as would be appreciated. Performance of image processingsystem 100 may be optimized using various cost functions andlearning/training algorithms, such as, but not limited to, Markovprocesses, convolution and pooling neural networks, genetic algorithms,and vector machine training as would be appreciated.

Various implementations of the invention provide an optimal set ofcamera settings at the onset of (and in some implementations during)acquisition of a high quality image, such as a three-dimensional imageof the face acquired by, for example, the lidar system. In someimplementations of the invention, acquisition of the high quality imagemay take significant time relative to a frame rate of camera 110. Forexample, the three-dimensional image of the face may take 1.5 seconds ormore to acquire, whereas the frame rate of camera 110 may be 30 framesper second (fps). Having the optimal set of camera settings at the onsetand having those camera settings updated during acquisition of the highquality image facilitates achieving optimal performance.

Various implementations of the invention facilitate detection of face(s)in image(s) 135 in a variety of conditions, including differing lightingconditions. Further, camera settings 155 can be adjusted and optimizedfor each face in image 135. In some implementations of the invention, afirst set of camera settings 155 may be optimized for a first face inthe uncontrolled environment and then a first image 135 (not necessarilya “high quality” one) may be captured. Next, a second set of camerasettings 155 may be optimized for a second face in the uncontrolledenvironment and then a second image 135 may be captured. This maycontinue until all faces in the uncontrolled environment are captured.Once all the faces are captured, face acquirer 350 may revisit each faceand acquire a high quality image of each face. In these implementations,optimized images of each face may be captured and then high qualityimages of some of the faces (due to time constraints of the targetsmoving in the scene) may be acquired.

In some implementations of the invention, once a face is detected inimage frames 135, a trajectory of the face is estimated in an effort topredict its expected location in subsequent image frames. Techniques forestimating motion of a target (i.e., a trajectory) in an image streamare well known. In such implementations, settings adjuster 220 mayadjust camera settings 155 based on the expected location of the face ineach subsequent image frame 135. Such implementations are particularlyuseful in environments when illumination of the face changes as thetarget moves in the scene.

While the invention has been described herein in terms of variousimplementations, it is limited only by the scope of the followingclaims, as would be apparent to one skilled in the art. These and otherimplementations of the invention will become apparent upon considerationof the disclosure provided above and the accompanying figures. Inaddition, various components and features described with respect to oneimplementation of the invention may be used in other implementations aswell.

What is claimed is:
 1. A method for controlling camera settings of acamera, the method comprising: receiving a first image from the camera,the first image captured by the camera at a first set of camerasettings; detecting a face in the first image; adjusting the camera to asecond set of camera settings based on the detected face, the second setof camera settings different from the first set of camera settings;receiving a second image from the camera, the second image captured bythe camera at the second set of camera settings; detecting the face inthe second image; determining a quality metric of the face in the secondimage, the quality metric indicative of an image quality of the face inthe second image; adjusting the camera to a new set of camera settingsto increase the quality metric of the face in new images, the new set ofcamera settings different from both the first set of camera settings andthe second set of camera settings; and acquiring the face upon reachinga sufficient quality metric of the face.
 2. The method of claim 1,wherein each of the sets of camera settings comprises an aperture, anexposure, and a gain.
 3. The method of claim 1, further comprising:detecting a windshield in an image; and changing a region of interest ofthe camera based on the detected windshield.
 4. The method of claim 3,further comprising adjusting the camera to a new set of camera settingsbased on the detected windshield.
 5. The method of claim 3, whereinchanging the region of interest of the camera comprises changing theregion of interest from a full frame region of interest to a windshieldregion of interest.
 6. The method of claim 1, further comprisingchanging a region of interest of the camera in response to detecting aface in the first image.
 7. The method of claim 6, wherein changing theregion of interest of the camera in response to detecting a face in thefirst image comprises changing the region of interest from a full frameregion of interest to a facial region of interest.
 8. The method ofclaim 1, further comprising, before receiving the first image from thecamera: iteratively adjusting the camera through a number of sets ofcamera settings, each set of camera settings tuned to a different one ofvarious scenarios that might exist in the uncontrolled environment. 9.The method of claim 1, wherein acquiring the face upon reaching asufficient quality metric of the face comprises acquiring athree-dimensional image of the face.
 10. A method for controlling camerasettings of a camera, the method comprising: receiving a first imagefrom the camera, the first image captured by the camera at a first setof camera settings; detecting a first event in the first image;adjusting the camera to a second set of camera settings based on thedetected event, the second set of camera settings different from thefirst set of camera settings; receiving a second image from the camera,the second image captured by the camera at the second set of camerasettings; detecting a second event in the second image; determining aquality metric associated with the second event in the second image, thequality metric indicative of a quality of the second event in the secondimage; adjusting the camera to a new set of camera settings to increasethe quality metric in new images, the new set of camera settingsdifferent from both the first set of camera settings and the second setof camera settings.
 11. A system for controlling camera settings of acamera comprising: a face detector configured to receive a first imagefrom the camera, the first image captured by the camera at a first setof camera settings, and configured to detect a face in the first image;a settings adjuster responsive to the face detector and configured toadjust the camera to a second set of camera settings based on thedetected face, the second set of camera settings different from thefirst set of camera settings, wherein the face detector is furtherconfigured to receive a second image from the camera, the second imagecaptured by the camera at the second set of camera settings, and furtherconfigured to detect the face in the second image; and a face recognizerconfigured to determine a quality metric of the face in the secondimage, the quality metric indicative of an image quality of the face inthe second image, wherein the settings adjuster is further configured toadjust the camera to a new set of camera settings to increase thequality metric of the face in new images, the new set of camera settingsdifferent from both the first set of camera settings and the second setof camera settings; and a face acquirer configured to acquire the faceupon reaching a sufficient quality metric of the face.
 12. The system ofclaim 11, wherein the settings adjuster is further configured toiteratively adjust the camera to new sets of camera settings while theface acquirer acquires the face.
 13. The system of claim 11, whereineach of the sets of camera settings comprises an aperture, an exposure,and a gain.
 14. The system of claim 11, wherein the face detector isfurther configured to detect a windshield in the first image and changea region of interest of the camera based on the detected windshield. 15.The system of claim 14, wherein the settings adjuster is furtherconfigured to adjust the camera to a new set of camera settings based onthe detected windshield.
 16. The system of claim 14, wherein changingthe region of interest of the camera comprises changing the region ofinterest from a full frame region of interest to a windshield region ofinterest.
 17. The system of claim 11, wherein the face detector isfurther configured to iteratively adjust the camera through a number ofpredetermined sets of camera settings until a face is detected in animage, each set of camera settings tuned to a different one of variousscenarios that might exist in the uncontrolled environment.